Contract testing using testrunner and evmone #276

mbenke · 2025-11-28T12:21:31Z

Contract testing (create contract and call its methods) using testrunner and evmone.
This PR integrates the C++ testrunner into the Nix build system and adds end-to-end integration tests to nix flake check (which is now run as part of the CI workflow).

Written using substantial Claude Code input, hence includes CLAUDE.md (which can also serve as a part of documentation).

Changes

Nix Build System:

Added Nix derivations for evmone dependencies (nix/evmone.nix, nix/intx.nix, nix/blst.nix)
Created testrunner package that builds the C++ test executor
Added checks.contests derivation that runs full pipeline integration tests
Uses nixpkgs packages (boost, nlohmann_json) instead of git submodules

Script Portability:

Made contest.sh configurable via environment variables with sensible defaults
Scripts work both in Nix builds (with overrides) and local development (with defaults)
Environment variables: SOLCORE_CMD, YULE_CMD, testrunner_exe, evmone

Testing:

Integration tests verify full pipeline: .solc → sol-core → .core → yule → .yul → solc → .hex → testrunner
nix flake check now runs contest test suite (6 tests)
Test results: basic.json (4 tests) and neg.json (2 tests) passing ✓

Documentation:

Added CLAUDE.md with architecture and Nix integration details
Updated README.md with testrunner build and usage instructions
Clear distinction between Haskell unit tests and integration tests

Usage

  # Run integration tests via Nix (recommended)
  nix flake check

  # Or build testrunner manually and run locally
  cmake -S . -B build && cmake --build build --target testrunner
  export testrunner_exe=build/test/testrunner/testrunner
  bash run_contests.sh

Allow overriding paths and commands via environment variables: - SOLCORE_CMD (default: cabal exec sol-core --) - YULE_CMD (default: cabal run yule --) - testrunner_exe (default: test/testrunner/testrunner) - evmone (default: ~/.local/lib/libevmone.so) This preserves the ability to run scripts outside of Nix builds while allowing Nix to override these values as needed.

Documents: - C++ testrunner architecture and components - Contest integration test system - Environment variable configuration - Nix build system and packages - Development workflow for local and Nix builds

Documents: - Building the C++ testrunner with cmake - Running integration tests manually and via Nix - Distinction between Haskell unit tests and integration tests - Test case structure and usage - Using contest.sh for individual test execution

Adds parallel 'integration-tests' job that runs 'nix flake check' to execute the C++ testrunner and contest integration tests on every PR. The workflow now has two parallel jobs: - build: Builds sol-core (existing) - integration-tests: Runs full pipeline tests (new)

Removes branch restriction from pull_request trigger so the workflow runs on all PRs regardless of target branch. Keeps push trigger restricted to main branch to avoid duplicate runs.

Replaces separate 'build' and 'integration-tests' jobs with a single 'build-and-test' job that runs 'nix flake check'. This is more efficient because: - nix flake check builds sol-core as a dependency (no duplicate builds) - Single job is simpler and clearer - Still runs full integration test suite

Y-Nak

I added two nits.
Also, currently, the tests fail because of the copyToMem implementation. But other than that, LGTM overall.

run_contests.sh

Y-Nak · 2026-01-12T11:01:54Z

contest.sh

+fi
+
+echo "Processing: $file"
+root_dir="$(cd "$(dirname "$(readlink --canonicalize "${BASH_SOURCE[0]}")")" && pwd)"


Suggested change

root_dir="$(cd "$(dirname "$(readlink --canonicalize "${BASH_SOURCE[0]}")")" && pwd)"

root_dir="$(cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")" && pwd)"

readlink -f doesn't work in macos(if coreutils are not installed). Probably realpath would be more portable.

readlink -f doesn't work in macos(if coreutils are not installed). Probably realpath would be more portable.

Does realpath work on macos without coreutils? On my (admittedly ancient - macOS 10.13) MacBook it is the oposite - system readlink and realpath from coreutils.
Is readlink an issue for you even when using nix develop?

We can always bemore paranoid and try

root_dir="$(realpath "$(dirname "${BASH_SOURCE[0]}")" || cd "$(dirname "$(readlink --canonicalize "${BASH_SOURCE[0]}")")" && pwd)"

realpath works on macos without coreutils (probably since around 10.15?). But nix develop and nix flake check works well with readlink --canonical. So it'd be fine to leave it as is for now.

clonker · 2026-01-12T12:07:33Z

CLAUDE.md

not sure how this is relevant to the pr, perhaps split it into a separate one?

clonker · 2026-01-12T12:21:38Z

CLAUDE.md

+**Packages** (`nix build .#<package>`):
+- `sol-core` - Main Haskell compiler
+- `testrunner` - C++ testrunner binary
+- `intx`, `blst`, `evmone` - EVM dependencies (built from source)


why though? intx and blst are pulled by evmone's cmake in a pinned version - the supported and tested combination, I would assume. Can't we just rely on that instead of potentially introducing version drift?

also i'm not sure what happens with nlohmann_json here, is Hunter completely disabled, which version does it take?

@clonker I am no expert on Nix, but from what I am told, in Nix builds, you're in a sandbox with no internet access. So Hunter can't download packages. That's why the current evmone.nix:

Disables Hunter completely (lines 24-32) by stubbing out the Hunter macros

Provides intx and blst explicitly from Nix instead

Can you suggest a better approach that works with Nix?

I can't, I'm not a Nix user at all; it just struck me as odd and potentially unstable / hard to maintain.

Well I think it's fine. We just shouldn't forget to update intx and blst together with evmone. I'm not sure about the specifics but perhaps we can encapsulate the intx, blst, and nlohmann_json deps in the evmone package so they're not top-level? They're not runtime dependencies but statically included in the binary.

mbenke · 2026-01-13T08:49:33Z

I added two nits. Also, currently, the tests fail because of the copyToMem implementation. But other than that, LGTM overall.

Which tests fail exactly? It seems that nix flake check succeeds on github - see https://github.com/argotorg/solcore/actions/runs/20598930090/job/59159651508?pr=276

Co-authored-by: Yoshitomo Nakanishi <yurayura.rounin.3@gmail.com>

Y-Nak

I suggested one change to be able to run nix flake check on macos.

flake.nix

Co-authored-by: Yoshitomo Nakanishi <yurayura.rounin.3@gmail.com>

Contract testing using testrunner and evmone

40b3771

mbenke requested review from clonker and d-xo November 28, 2025 12:21

flake.nix: add boost and cmake

256702d

mbenke mentioned this pull request Dec 10, 2025

Make testrunner buildable on Ubuntu noble LTS #274

Closed

mbenke added 7 commits December 29, 2025 13:35

contract tests - small updates

697b928

contract testing scripts

a91b66e

Add DEBUG output to contest.sh for troubleshooting

de8be5d

Remove DEBUG output from contest.sh

9f416b3

Add CLAUDE.md documentation for C++ testrunner and Nix integration

14e0f36

Documents: - C++ testrunner architecture and components - Contest integration test system - Environment variable configuration - Nix build system and packages - Development workflow for local and Nix builds

mbenke marked this pull request as ready for review December 29, 2025 16:26

mbenke added 5 commits December 29, 2025 17:33

Run CI workflow on PRs to any branch

f2c4dbf

Removes branch restriction from pull_request trigger so the workflow runs on all PRs regardless of target branch. Keeps push trigger restricted to main branch to avoid duplicate runs.

goevmlab.nix: update hash

15346a7

flake.nix: hevm 0.56 does not need patching

0bbeb66

Y-Nak self-requested a review January 6, 2026 13:28

Y-Nak reviewed Jan 12, 2026

View reviewed changes

clonker reviewed Jan 12, 2026

View reviewed changes

run_contests.sh: set -euo pipefail

223e1b0

Co-authored-by: Yoshitomo Nakanishi <yurayura.rounin.3@gmail.com>

Y-Nak reviewed Jan 13, 2026

View reviewed changes

flake.nix Outdated Show resolved Hide resolved

more portable evmone setting

d89bc7e

Co-authored-by: Yoshitomo Nakanishi <yurayura.rounin.3@gmail.com>

Base automatically changed from issue261 to mbenke/canstore January 13, 2026 15:18

Base automatically changed from mbenke/canstore to main January 13, 2026 15:21

mbenke force-pushed the mbenke/contest branch from 11d39b7 to d89bc7e Compare January 13, 2026 17:51

Merge branch 'main' into mbenke/contest

df737d2

mbenke force-pushed the mbenke/contest branch from be6f4ef to df737d2 Compare January 14, 2026 14:22

	root_dir="$(cd "$(dirname "$(readlink --canonicalize "${BASH_SOURCE[0]}")")" && pwd)"
	root_dir="$(cd "$(dirname "$(realpath "${BASH_SOURCE[0]}")")" && pwd)"

Contract testing using testrunner and evmone #276

Are you sure you want to change the base?

Contract testing using testrunner and evmone #276

Uh oh!

Conversation

mbenke commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Nix Build System:

Script Portability:

Testing:

Documentation:

Usage

Uh oh!

Y-Nak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbenke Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mbenke commented Jan 13, 2026

Uh oh!

Y-Nak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mbenke commented Nov 28, 2025 •

edited

Loading

mbenke Jan 13, 2026 •

edited

Loading